logo-stackoverflow.png

Stack Exchange Data Dump

by: Stack Exchange, Inc.

Topics: Stack Exchange Data Dump - 2023-03-08

Contributor: Stack Exchange Community

This is an anonymized dump of all user-contributed content on the Stack Exchange network. Each site is formatted as a separate archive consisting of XML files zipped via 7-zip using bzip2 compression. Each site archive includes Posts, Users, Votes, Comments, PostHistory and PostLinks. For complete schema information, see this meta post. All user content contributed to the Stack Exchange network is cc-by-sa 4.0 licensed, intended to be shared and remixed. We even provide all our data as a convenient data dump.

License: https://creativecommons.org/licenses/by-sa/4.0/

But our cc-by-sa 4.0 licensing, while intentionally permissive, does require attribution:

Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
Specifically the attribution requirements are as follows:

Visually display or otherwise indicate the source of the content as coming from the Stack Exchange Network. This requirement is satisfied with a discreet text blurb, or some other unobtrusive but clear visual indication.
Ensure that any Internet use of the content includes a hyperlink directly to the original question on the source site on the Network (e.g., http://stackoverflow.com/questions/12345)
Visually display or otherwise clearly indicate the author names for every question and answer used
Ensure that any Internet use of the content includes a hyperlink for each author name directly back to his or her user profile page on the source site on the Network (e.g., http://stackoverflow.com/users/12345/username), directly to the Stack Exchange domain, in standard HTML (i.e. not through a Tinyurl or other such indirect hyperlink, form of obfuscation or redirection), without any “nofollow” command or any other such means of avoiding detection by search engines, and visible even with JavaScript disabled.

For more information, see the Stack Exchange Terms of Service.

Addeddate: 2014-01-21 18:54:32

Identifier: stackexchange

plus-circle Add Review

comment
Reviews

Reviewer: Vahid887 - - March 7, 2023
Subject: Not updated yet

The lastest data is still for December.

Reviewer: jesan2021 - - February 21, 2023
Subject: stackoverflow.com-Posts.7z file is missing

I cannot find the stackoverflow.com-Posts.7z file. Am I missing something?

Reviewer: Aleksandar Jeftić - - October 10, 2022
Subject: When I try to unarchive it, I get error "Is not archive"

Then I tried solutions from below and not working, not sure if I am missing something, or files are bad?

Reviewer: mikeeus - - October 7, 2022
Subject: Clubmed you're a life saver

If anyone is struggling with opening the 7z files look at the review below. This beautiful angel saved me a massive headache trying to get these archives unzipped.

Reviewer: clubmed - - October 6, 2022
Subject: Corrupt archive/data error issues

I was running into the same corrupt archive issues that others have mentioned, regardless of the way in which I downloaded the files (tried browser, the IA CLI utility, wget and the torrent file). Not sure what the issue was, but there were three lines being added to the beginning of the archives before the 7z archive header proper starts, that include the "Content-Disposition" header. I was able to fix the archives by opening the files in VIM in binary mode and deleting the first three lines, but it can also be done with a single line of bash:
FILE=FILENAME; head -3 $FILE | wc -c | xargs -I{} dd if=$FILE bs={} skip=1 conv=notrunc of=fixed-$FILE

Just replace FILENAME with the file you're working with and you should be good to go.

Reviewer: SkobelevIgor - - December 19, 2020
Subject: Variant of possible schema (pg & mysql )

Here is a variant of possible schema: https://github.com/SkobelevIgor/stackexchange-xml-converter/tree/main/schema_example Hope that helps!

Reviewer: jean_1mm - - July 22, 2020
Subject: Can someone tell me the table structure of Posts、Votes?

Can someone tell me the table structure of Posts、Votes、Badegs、Comments、Users、Tags? thansks

Reviewer: andreymal - - October 1, 2019
Subject: License

Don't trust the description. The license is actually CC BY-SA 3.0 because Stack Exchange changed it without explicit permission from original authors. Read more: https://meta.stackexchange.com/questions/333089

Reviewer: Tim_Nagy - - September 25, 2019
Subject: Old dump

I downloaded the Stack Overflow posts dump ('stackoverflow.com-Posts.7z') on 13 Sept 2019 which is lastly modified on 4 Sept 2019 according to https://archive.org. However, I found the latest post in the dump with date 11-03-2018 which does not make sense to me. I am not sure what the modified date of the dump means?

I am looking forward to your replies.

Reviewer: Habit Lin - - September 5, 2019
Subject: Meaning of some links in stackoverflow.com-PostLinks.7z.

Can anyone tell me the meaning of those links whose type is 'Linked'?

For instance, if there is a 'Linked' type edge between question A and question B, what information can we get from this relationship? Does it mean that question A references question B? Or it means something else?

Thanks.

Reviewer: abhishekverma3007 - - May 20, 2019
Subject: Spark project to review all stackoverflow data

https://github.com/prompt-spark/stackexchange-spark-scala-analyser

Reviewer: hearrywang - - October 17, 2018
Subject: information about data

I want to know every type of attribute in stackoverflow.com-Comments.7z. I did not find the link of this specific information.

Reviewer: jickerson - - July 29, 2018
Subject: No files available

Similar to the previous comment, I also am unable to download the files despite being logged in. Is the archive no longer available?

Reviewer: RamtinYazdanian - - February 20, 2018
Subject: Access restricted

Despite my having signed in, access to virtually all files (except maybe 5 of them) is restricted and I have no idea why.

Reviewer: nederlandsmeisje - - February 20, 2018
Subject: XML file into Stata

Did some of you guys manage to get the XML files into Stata? I really want to know how I can tackle this.

Reviewer: optimus9p - - February 19, 2018
Subject: Not available for download

The dataset cannot be downloaded even though I am logged into archive.org.

Reviewer: artoog2 - - November 5, 2017
Subject: Data Error

I tried downloading this 4 times today and each time I'm getting a "Data Error" while extracting the Posts.xml file through 7zip. Can someone post the checksum for the Posts archive?

Reviewer: Muldones - - October 20, 2017
Subject: stackoverflow.com-PostLinks.7z corrupted

The file stackoverflow.com-PostLinks.7z seems to be corrupted. I can not unzip it...

Reviewer: aNeutrino - - July 14, 2017
Subject: File size limit

Hi :)
I can not use torrent (I am in Cuba)
The only option for us is to download file with wget.
However, when we try to download full stack exchange data dump we have a message:
"total size of requested files (44 GB) is too large for zip-on-the-fly"

Can I ask please to remove this limitation?

Reviewer: amz3 - - July 9, 2017
Subject: Excellent

This dump was used to generate offline static version of stackoverflow websites as part of the kiwix.org project.

http://download.kiwix.org/zim/stack_exchange/

Reviewer: Venkatesh Prasad - - April 21, 2017
Subject: Great data set

My first time dabbling with this data set and I am having fun. Thanks for using a simple line-by-line XML format :)

Reviewer: JasonC3 - - April 1, 2017
Subject: Documentation

Being discussed at https://meta.stackexchange.com/questions/293127. A list of sites and their corresponding database filenames can be found there as well.

Also note: Info about sites and their corresponding file names can be indirectly obtained via the /sites API query (http://api.stackexchange.com/docs/sites): For every site, the file name will match either the primary URL of that site *or* the URL of one of that site's aliases. There are no exceptions beyond SO (whose dump is uniquely split among multiple files).

Sites that are very new relative to the archive date are not included in the dump.

Reviewer: coder_chenzhi - - March 24, 2017
Subject: Incomplete

Why no softwareengineering.stackexchange?

Reviewer: fturco - - March 5, 2017
Subject: Better filenames

Let me suggest you to add dates in YYYY-MM-DD or YYYYMMDD format into the filenames. For example: stackexchange-20161215/3dprinting.stackexchange.com.7z or stackexchange/3dprinting.stackexchange.com.20161215.7z

Reviewer: Mooash - - June 28, 2016
Subject: Torrent out of date

Looks like the torrent is out of date again, I'm stuck at 800MB whilst it continually tries to download files.

Reviewer: Greg Lindahl - - March 18, 2016
Subject: .torrent is fixed

The former limitation of 25 gigabytes for a torrent have been relaxed for this item, and the torrent is again working.

If you want to parse these large XML files, you need to use a streaming parser. I'm not surprised that an XML editor would have problems with a 40 gigabyte XML file! Most XML parsing software libraries have a streaming option.

Reviewer: vishal14 - - March 5, 2016
Subject: How to open posts.xml file?

I have downloaded and extracted posts.xml file of stackoverflow. Size of the file is around 40 GB and I am not able to open it in xml editors. Can someone please suggest how to open or parse this huge file?

Reviewer: shankar321 - - January 11, 2016
Subject: What sense do we make of the files?

I see over 300 files, whereas there are only 6 XML files (large ones) as far as I remember.

I cannot download the torrent version either - can someone help make sense of the files?

Shankar

Reviewer: jlewi - - December 30, 2015
Subject: torrent out of date

The bit torrent link:
https://archive.org/download/stackexchange/stackexchange_archive.torrent

appears to link to an older version of the data dump. For stackoverflow the latest post I saw was from 2014.

However, if the .zip file
https://archive.org/compress/stackexchange/formats=7Z&file=/stackexchange.zip

appears to have data upto august 2015.

Reviewer: Jesse_W - - November 6, 2015
Subject: This hit the shuffle.php bug

And that broke the webseeds (see https://archive.org/post/1047899 ). This review should hopefully fix the issue.

Reviewer: D1Doris - - September 4, 2015
Subject: stackexchange_meta.sqlite

StevenLJohnson, I have the file and can open it without any problems. What exactly do you mean by "unable to access"? If you mean that you don't have the file, I can send it to you.

Reviewer: StevenLJohnson - - August 31, 2015
Subject: Missing File

I am unable to access the file stackexchange_meta.sqlite

Does anyone know of a source for this file?

Reviewer: xcombelle - - July 21, 2015
Subject: for those which can't make bittorent working

There is some reasons for it https://archive.org/about/faqs.php#Archive_BitTorrents

Reviewer: timiblossom - - July 2, 2015
Subject: Bittorrent download broken

It stopped at around 70% for a couple of days and could never move forward.

Reviewer: sathvik - - May 22, 2015
Subject: Thanks for sharing

Thanks for sharing the community data. It will greatly benefit research groups.

Reviewer: dmpetrov - - May 3, 2015
Subject: April data

Great data set. Thank you for sharing.

I see only March data. How can I get April data?
What about January and February?

Thanks,
Dmitry

Reviewer: alisa1 - - April 10, 2015
Subject: Resolved!

I also tried couple of times. It was failed at the same point.
But then I tried when I logged in, and I was able to download the whole file :-)

Reviewer: big_t_dub - - April 3, 2015
Subject: 70%

stuck at 70.7% download complete via utorrent- arg!

this should be made avail via ftp!!!

Reviewer: Ihor Bobak - - March 27, 2015
Subject: File is broken

At the top right corner of this page there is a link to zip archive. I've downloaded it twice (on different machines, in different countries). The file was always broken.

Torrent stucks on 70.8%.

Can anyone help to get this file?

Reviewer: gnijuohz - - March 25, 2015
Subject: No seed?

It stopped at around 70%.

Reviewer: klitzkrieg - - March 24, 2015
Subject: Seeds for 3/16/15 version?

Everybody's stuck at 70.7%

Reviewer: Nemo_bis - - February 7, 2015
Subject: Thanks and tests

Thanks for the September update, eager to see the next one. Did someone try importing this data into a StackExchange instance?

Fun to see how small the whole SE network is after all, only few GB compressed. Wikimedia projects dumps compress very well too, but they're still much bigger (while fitting a common hard disk anyway!).

Reviewer: shamsazad - - August 19, 2014
Subject: Latest Dump.

When will be latest dump from stackoverflow will be posted over here.

Reviewer: Jenson555 - - July 26, 2014
Subject: Really Cool

This is an Awesome Stuff..Cheers..:)

294,330 Views

28 Favorites

44 Reviews